<!DOCTYPE html>
<html>
<head>
<title>周志华《机器学习》课后习题解答系列（三）：Ch2 - 模型评估与选择</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css">
/* GitHub stylesheet for MarkdownPad (http://markdownpad.com) */
/* Author: Nicolas Hery - http://nicolashery.com */
/* Version: b13fe65ca28d2e568c6ed5d7f06581183df8f2ff */
/* Source: https://github.com/nicolahery/markdownpad-github */

/* RESET
=============================================================================*/

html, body, div, span, applet, object, iframe, h1, h2, h3, h4, h5, h6, p, blockquote, pre, a, abbr, acronym, address, big, cite, code, del, dfn, em, img, ins, kbd, q, s, samp, small, strike, strong, sub, sup, tt, var, b, u, i, center, dl, dt, dd, ol, ul, li, fieldset, form, label, legend, table, caption, tbody, tfoot, thead, tr, th, td, article, aside, canvas, details, embed, figure, figcaption, footer, header, hgroup, menu, nav, output, ruby, section, summary, time, mark, audio, video {
  margin: 0;
  padding: 0;
  border: 0;
}

/* BODY
=============================================================================*/

body {
  font-family: Helvetica, arial, freesans, clean, sans-serif;
  font-size: 14px;
  line-height: 1.6;
  color: #333;
  background-color: #fff;
  padding: 20px;
  max-width: 960px;
  margin: 0 auto;
}

body>*:first-child {
  margin-top: 0 !important;
}

body>*:last-child {
  margin-bottom: 0 !important;
}

/* BLOCKS
=============================================================================*/

p, blockquote, ul, ol, dl, table, pre {
  margin: 15px 0;
}

/* HEADERS
=============================================================================*/

h1, h2, h3, h4, h5, h6 {
  margin: 20px 0 10px;
  padding: 0;
  font-weight: bold;
  -webkit-font-smoothing: antialiased;
}

h1 tt, h1 code, h2 tt, h2 code, h3 tt, h3 code, h4 tt, h4 code, h5 tt, h5 code, h6 tt, h6 code {
  font-size: inherit;
}

h1 {
  font-size: 28px;
  color: #000;
}

h2 {
  font-size: 24px;
  border-bottom: 1px solid #ccc;
  color: #000;
}

h3 {
  font-size: 18px;
}

h4 {
  font-size: 16px;
}

h5 {
  font-size: 14px;
}

h6 {
  color: #777;
  font-size: 14px;
}

body>h2:first-child, body>h1:first-child, body>h1:first-child+h2, body>h3:first-child, body>h4:first-child, body>h5:first-child, body>h6:first-child {
  margin-top: 0;
  padding-top: 0;
}

a:first-child h1, a:first-child h2, a:first-child h3, a:first-child h4, a:first-child h5, a:first-child h6 {
  margin-top: 0;
  padding-top: 0;
}

h1+p, h2+p, h3+p, h4+p, h5+p, h6+p {
  margin-top: 10px;
}

/* LINKS
=============================================================================*/

a {
  color: #4183C4;
  text-decoration: none;
}

a:hover {
  text-decoration: underline;
}

/* LISTS
=============================================================================*/

ul, ol {
  padding-left: 30px;
}

ul li > :first-child, 
ol li > :first-child, 
ul li ul:first-of-type, 
ol li ol:first-of-type, 
ul li ol:first-of-type, 
ol li ul:first-of-type {
  margin-top: 0px;
}

ul ul, ul ol, ol ol, ol ul {
  margin-bottom: 0;
}

dl {
  padding: 0;
}

dl dt {
  font-size: 14px;
  font-weight: bold;
  font-style: italic;
  padding: 0;
  margin: 15px 0 5px;
}

dl dt:first-child {
  padding: 0;
}

dl dt>:first-child {
  margin-top: 0px;
}

dl dt>:last-child {
  margin-bottom: 0px;
}

dl dd {
  margin: 0 0 15px;
  padding: 0 15px;
}

dl dd>:first-child {
  margin-top: 0px;
}

dl dd>:last-child {
  margin-bottom: 0px;
}

/* CODE
=============================================================================*/

pre, code, tt {
  font-size: 12px;
  font-family: Consolas, "Liberation Mono", Courier, monospace;
}

code, tt {
  margin: 0 0px;
  padding: 0px 0px;
  white-space: nowrap;
  border: 1px solid #eaeaea;
  background-color: #f8f8f8;
  border-radius: 3px;
}

pre>code {
  margin: 0;
  padding: 0;
  white-space: pre;
  border: none;
  background: transparent;
}

pre {
  background-color: #f8f8f8;
  border: 1px solid #ccc;
  font-size: 13px;
  line-height: 19px;
  overflow: auto;
  padding: 6px 10px;
  border-radius: 3px;
}

pre code, pre tt {
  background-color: transparent;
  border: none;
}

kbd {
    -moz-border-bottom-colors: none;
    -moz-border-left-colors: none;
    -moz-border-right-colors: none;
    -moz-border-top-colors: none;
    background-color: #DDDDDD;
    background-image: linear-gradient(#F1F1F1, #DDDDDD);
    background-repeat: repeat-x;
    border-color: #DDDDDD #CCCCCC #CCCCCC #DDDDDD;
    border-image: none;
    border-radius: 2px 2px 2px 2px;
    border-style: solid;
    border-width: 1px;
    font-family: "Helvetica Neue",Helvetica,Arial,sans-serif;
    line-height: 10px;
    padding: 1px 4px;
}

/* QUOTES
=============================================================================*/

blockquote {
  border-left: 4px solid #DDD;
  padding: 0 15px;
  color: #777;
}

blockquote>:first-child {
  margin-top: 0px;
}

blockquote>:last-child {
  margin-bottom: 0px;
}

/* HORIZONTAL RULES
=============================================================================*/

hr {
  clear: both;
  margin: 15px 0;
  height: 0px;
  overflow: hidden;
  border: none;
  background: transparent;
  border-bottom: 4px solid #ddd;
  padding: 0;
}

/* TABLES
=============================================================================*/

table th {
  font-weight: bold;
}

table th, table td {
  border: 1px solid #ccc;
  padding: 6px 13px;
}

table tr {
  border-top: 1px solid #ccc;
  background-color: #fff;
}

table tr:nth-child(2n) {
  background-color: #f8f8f8;
}

/* IMAGES
=============================================================================*/

img {
  max-width: 100%
}
</style>
</head>
<body>
<h2>本章概要</h2>
<p>本章讲述了模型评估与选择（model evaluation and selection）的相关知识：</p>
<p>2.1 经验误差与过拟合（empirical error &amp; overfitting）</p>
<blockquote>
<p>精度accuracy、训练误差（经验误差）training error（empirical error）、<strong>泛化误差</strong>generalization error、<strong>过拟合</strong>overfitting、欠拟合underfitting；</p>
</blockquote>
<p>2.2 模型评估方法（evaluate method）</p>
<blockquote>
<p>测试误差testing error、留出法hold-out、分层采样stratified sampling、交叉验证法cross validation、<strong>k-折交叉验证</strong>k-fold cross validation、留一法leave-one-out（LOO）、<strong>自助法<strong>bootstrapping、自助采样bootstrap sampling、包外估计out-of-bag estimate、</strong>调参</strong>parameter tuning、验证集validation set；</p>
</blockquote>
<p>2.3 模型性能度量（performance measure）</p>
<blockquote>
<p>错误率error rate、查准率（准确率）precision、查全率（召回率）recall、P-R曲线、平衡点BEP、<strong>F1/Fβ</strong>、<strong>混淆矩阵<strong>、</strong>ROC曲线</strong>、AUC、代价敏感cost-sensitive、<strong>代价矩阵</strong>cost matrix、代价曲线cost curve、期望总体代价；</p>
</blockquote>
<p>2.4 模型比较检验（comparation &amp; testing）</p>
<blockquote>
<p>假设检验hypothesis test、拒绝假设、t-检验t-test、Friedman检验、后续检验post-hoc test、Friedman检验图；</p>
</blockquote>
<p>2.5 偏差与方差（bias &amp; variance）</p>
<blockquote>
<p>偏差-方差窘境bias-variance dilemma；</p>
</blockquote>
<h2>习题解答</h2>
<h3>2.1 分层抽样划分训练集与测试集</h3>
<blockquote>
<p><img src="Ch2/2.1.png" /></p>
</blockquote>
<p>根据分层采样原则，共有方法： </p>
<p><img src="Ch2/2.1.1.png" />.</p>
<hr />
<h3>2.2 留一法与k-折交叉验证法比较</h3>
<blockquote>
<p><img src="Ch2/2.2.png" /></p>
</blockquote>
<p>因为测试集被划分到训练样本中多的类，设一共100个样本：</p>
<p>留一法：测试集1个样本，训练集99个样本且有50个与测试集真实类别不同，故测试集无法被划分到正确的类，错误率<strong>100%</strong>；</p>
<p>交叉验证法：在采用分层抽样的前提下，分类靠随机猜，错误率因为<strong>50%</strong>； </p>
<hr />
<h3>2.3 F1值与BEP的关联</h3>
<blockquote>
<p><img src="Ch2/2.3.png" /></p>
</blockquote>
<p>首先给出度量定义：</p>
<ul>
<li>
<p>BEP：是P-R曲线上的平衡点坐标值，BEP = P = R (即准确率 = 召回率)；</p>
</li>
<li>
<p>F1值：是P与R的调和平均，1/F1 = (1/P + 1/R) / 2;</p>
</li>
</ul>
<p>所以 BEP = F1 (当P = R时) -&gt; BEP(A) &gt; BEP(B).</p>
<hr />
<h3>2.4 TPR、FPR、P、R之间的关联</h3>
<blockquote>
<p><img src="Ch2/2.4.png" /></p>
</blockquote>
<p>给出混淆矩阵示例如下：</p>
<blockquote>
<p><img src="Ch2/2.4.1.png" /></p>
</blockquote>
<p>然后给出各度量的定义式：</p>
<blockquote>
<p><img src="Ch2/2.4.2.png" /></p>
</blockquote>
<p>详细解释是：</p>
<ul>
<li>P，查准率（准确率），（预测正例）中（真实正例）的比例.</li>
<li>R，查全率（召回率），（真实正例）中（预测正例）的比例.</li>
<li>TPR，真正例率，（预测正例）中（真实正例）的比例，TPR = P.</li>
<li>FPR，假正例率，（真实反例）中（预测正例）的比例.</li>
</ul>
<hr />
<h3>2.5 AUC推导（有限样例下）</h3>
<blockquote>
<p><img src="Ch2/2.5.png" /></p>
</blockquote>
<p>直接给出大致思路如下图：</p>
<p><img src="Ch2/2.5.1.jpg" /></p>
<hr />
<h3>2.6 错误率与ROC曲线的关系</h3>
<blockquote>
<p><img src="Ch2/2.6.png" /></p>
</blockquote>
<p>错误率可由代价-混淆矩阵得出；</p>
<p>ROC曲线基于TPR与FPR表示了模型在不同截断点取值下的泛化性能。</p>
<p>ROC曲线上的点越靠近（1，0）学习器越完美，但是常需要通过计算等错误率来实现P、R的折衷，而P、R则反映了我们所侧重部分的错误率。</p>
<hr />
<h3>2.7 ROC曲线与代价曲线的对应关系</h3>
<blockquote>
<p><img src="Ch2/2.7.png" /></p>
</blockquote>
<p>ROC曲线的点对应了一对（TPR,FPR），即一对（FNR,FPR），由此可得一条代价线段（0,FPR)--(1,FNR），由所有代价线段构成簇，围取期望总体代价和它的边界--代价曲线。所以说，ROC对应了一条代价曲线，反之亦然。</p>
<hr />
<h3>2.8 ROC曲线与代价曲线的关系</h3>
<blockquote>
<p><img src="Ch2/2.8.png" /></p>
</blockquote>
<p>比较见表：</p>
<table>
<thead>
<tr>
	<th>Max-min</th>
	<th>z-score</th>
</tr>
</thead>
<tbody>
<tr>
	<td>方法简单</td>
	<td>计算量相对大一些</td>
</tr>
<tr>
	<td>容易受高杠杆点和离群点影响</td>
	<td>对离群点敏感度相对低一些</td>
</tr>
<tr>
	<td>当加入新值超出当前最大最小范围时重新计算所有之前的结果</td>
	<td>每加入新值都要重新计算所有之前结果</td>
</tr>
</tbody>
</table>
<hr />
<h3>2.9 卡方检验过程</h3>
<blockquote>
<p><img src="Ch2/2.9.png" /></p>
</blockquote>
<p>可直接参考：<a href="http://baike.baidu.com/link?url=G2Md2m54oqVpFaSGDFjZJ4myZ9EHw8m2HSUx6wgpgE1MnePNrATY-GPjrQD62aRhu4StvgOmaM0xe8alOuyV3uBu3PLJDq2aeo29nz_P4OBE4d4YTLlIkQ3eelB5ByiP">卡方检验 - 百度百科</a></p>
<hr />

</body>
</html>
<!-- This document was created with MarkdownPad, the Markdown editor for Windows (http://markdownpad.com) -->
